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Background 
The consultation on REF 2021 is open until October 15t" 2018. Two documents have been 
published by the REF team: Draft Guidance and Draft Panel Criteria and Working Methods. 


We recently convened a workshop to share our draft response to the guidance about public 
engagement in the draft documents. The event provided very helpful feedback and advice 
from sector representatives, and the views and comments expressed at the event have 
informed the response we lay out here. The briefing paper and write up of the consultation 
event are available separately. 


Our goal in this response is to make practical suggestions about what (if anything) could be 
changed or added to improve the guidance. By improve, we mean: 


e Make it clearer 

e Make it more coherent and consistent 

e Make it more ‘appropriate’ (a term used consistently in the consultation. We are 
interpreting ‘appropriate’ to mean in accordance with good practice, and realistic 
within the constraints of the REF exercise) 


In fact, there are very few explicit mentions of public engagement in the two documents, 
although what is included offers strong encouragement to universities to consider including 
impacts arising from public engagement in their submitted case studies. We include these 
references in Annex 1. 


The purpose of this document is to share the key points we plan to make in our consultation 
response. The template for responses limits answers to just 300 words. This document also 
provides a more detailed rationale for our responses, and fleshes out some of our 
recommendations in more detail. For instance, we are suggesting some significant changes 
be made to Annex A of the Draft Criteria (Areas of Impact). 


The consultation questions 
In total, the consultation asks 24 questions. We are only responding to 9 of these questions, 
pertinent to public engagement: 


Questions about the Draft Guidance 

1. Is the guidance clear in 'Part 1: Overview of the assessment framework‘? 

2. Is the guidance clear in 'Part 2: Submissions' 

13. Is the guidance in 'Part 3, Section 3: Impact’ clear? 

15. Is the guidance in 'Part 3, Section 5: Environment’ clear? 

16. Please provide any further comments on the ‘Guidance on submissions’, including 
Annexes A-M 


Questions about the Draft Criteria and working methods 

4. Are the criteria clear and appropriate in 'Part 3, Section 3: Impact‘? 

5. Are the criteria clear and appropriate in 'Part 3, Section 5: Environment‘? 

7. Are the criteria clear and appropriate in ‘Part 5: Panel working methods‘? 

8. Overall, the ‘Panel criteria and working methods’ achieves an appropriate balance 
between consistency and allowing for discipline-based differences between the 
panels 


Previous consultations and reviews of PE in the REF 
The NCCPE has provided input to inform the framing of public engagement in the REF, over 
a number of years. You can access an overview and links on the NCCPE website. Of 


particular note are: 


The NCCPE’s review of Public Engagement in the 2014 REF, which analysed the 
database of impact case studies and impact templates to examine how public 


engagement featured and to draw oud lessons and recommendations to inform REF 
2021 

The NCCPE’s response to the 2017 consultation on the REF 

The briefing paper and write up of a workshop hosted by the REF team in February 
2018 which explored how public engagement should feature in the guidance for REF 
2021 

The briefing paper and write up of the consultation event we ran in September 2018, 
which has informed this response 


Headlines of our response 

While broadly supportive of the guidance and the continuity achieved with REF 2014, we 
identify a small number of significant issues which we believe require attention. In 
particular: 


The guidance should highlight the importance of considering Equality and Diversity 
in relation to impact, not just internal HEI staffing policies. There should be a clear 
expectation set by the panels that equality and diversity be addressed in impact 
planning (in the ‘environment’), and, where relevant, within impact case studies. We 
recommend that the 2010 Equality Act and the Public Duty regarding socio- 
economic inequalities should be referenced to provide an appropriate framing of 
this expectation. 


e While we are pleased to see that impacts arising from public engagement are 
encouraged, we suggest various ways in which this expectation could be made more 
explicit and coherent, to encourage a more strategic and comprehensive response 
from HEls. We recommend changes to the framing of the areas and indicators of 
impact (Annex A of the Draft Criteria and Working Methods) to ensure that impacts 
arising from public engagement are considered more holistically (for instance in 
relation to influencing policy making and professional practice, not just in terms of 
individual learning outcomes). 

e Agreater emphasis on evaluation is vital if we want to see a step change in the 
quality of submissions and the robustness of evidence submitted in 2021. We 
recommend that this is signalled more explicitly. 

e While recognising that the REF is focused on assessing the outcomes and impacts 
arising from research (not the engagement process itself) we recommend that more 
clarity is offered about how ‘pathways’ to impact will be judged by panels as part of 
the assessment process. It is clear from talking to panellists that these do contribute 
contextual evidence to inform their judgements, and this should be explained more 
explicitly. 

e Given that one of the core purposes of the REF is to ‘create a strong performance 
incentive for HEls and individual researchers’ (Guidance on Submissions, p.9) we also 
recommend that the guidance encourages HEls to consider active user engagement 
in the research process (for instance in relation to approaches to Open Access), in 
line with good practice. 

e We argue for a minimum of three assessors (at least two research users) for each 
impact case study, and for more clarity on how the evidence from the Environment 
section will be used to inform the assessment of impact case studies. 


The consultation limits responses to 300 words, which means very little rationale or detail 
can be included. Therefore, as well as including our consultation responses to key questions, 
we provide background notes to offer more context where we feel that will be helpful. 


If you have comments on this response, please contact paul. manners@uwe.ac.uk. 


We would be delighted if you chose to reference our comments in your consultation 
response. 


CONSULTATION RESPONSE 


DRAFT GUIDANCE ON SUBMISSIONS 


Question 1a. Guidance on submissions: Part 1: Overview of the assessment framework: 
is the guidance clear? 


In our response to this question we argue two points: that the principles of equity and 
equality, laid out in paragraph 27, should be refined to clarify the panels’ commitment to 
treating all types of impact equally, and to widen the treatment of Equality and Diversity 
beyond research careers. 


Question 1A 

Equity (27a) is defined exclusively in terms of research outputs and the commitment to 
judging all types of outputs in an equitable way. This principle should be extended to the 
‘types of impact’ as well. This will affirm the clear guidance elsewhere (e.g. 273, Draft 
Criteria: The main panels wish to encourage the submission of a broad range of types of 
impact). It also addresses the recommendation in the Stern review that impact be 
broadened and deepened (82-84). We suggest the following changes (new wording 
underlined): 


Equity: All types of research and all forms of research output and impact across all 
disciplines shall be assessed on a fair and equal basis. Panels [] recognise and treat on an 
equal footing excellence in research and impact across the spectrum... The REF aims to 
assess all types of research and impact... 


Equality (27b) is defined exclusively in terms of ‘internal’ codes of practice regarding 
processes of submitting staff. It is important that E&D issues are also addressed in 
relation to impact (which includes ‘the reduction or prevention of harm, risk, cost of other 
negative effects’). Impact generating activities fall squarely under the 2010 Equality Act, 
and in particular the Public Sector Equality Duty which requires public bodies to take a 
proactive and anticipatory approach to promoting equality. We suggest these changes to 
the wording: 

Equality: HEIs are strongly encouraged to embed equality and diversity, and are expected 
to comply with equality legislation, regarding their processes for submitting staff and 
outputs and in their approach to achieving impact. 


In our response to consultation question 15a we suggest changes to the environment and 
impact case study templates to incorporate references to E&D. 


We recommend that the section on E&D (51 — 55) is re-named ‘E&D in Research Careers’ 
to clarify its focus. 


Background notes to Qla 


While there are clear ethical reasons for considering equality and diversity in relation to 
impact (for instance, many researchers are committed to working with ‘underserved’ 
audiences, and to addressing inequalities), there are also legal requirements to take account 
of. The Equality Act 2010 includes a Public sector duty regarding socio-economic 
inequalities 


An authority to which this section applies must, when making decisions of a strategic 
nature about how to exercise its functions, have due regard to the desirability of 
exercising them in a way that is designed to reduce the inequalities of outcome which 
result from socio-economic disadvantage (P.1). 


It also requires that public bodies have due regard to the need to: 
e eliminate discrimination 
e advance equality of opportunity 
e foster good relations between different people when carrying out their activities 


The Equality Duty applies across Great Britain to the public bodies listed in Schedule 19 (as 
amended), which includes HEls and HEFCE, and to any other organisation when it is carrying 
out a public function. 


It identifies the following ‘protected characteristics’: 


Age; disability; gender reassignment; marriage and civil partnership; pregnancy and 
maternity; race; religion or belief; sex; sexual orientation (P.4) 


It spells out the expectations in more detail in section 149: 
Public sector equality duty 


(1) A public authority must, in the exercise of its functions, have due regard to the need 
to— 


(a) eliminate discrimination, harassment, victimisation and any other conduct 
that is prohibited by or under this Act; 


(b) advance equality of opportunity between persons who share a relevant 
protected characteristic and persons who do not share it; 


(c) foster good relations between persons who share a relevant protected 
characteristic and persons who do not share it. 


(3) Having due regard to the need to advance equality of opportunity between persons who 
share a relevant protected characteristic and persons who do not share it involves having 
due regard, in particular, to the need to— 


(a) remove or minimise disadvantages suffered by persons who share a relevant 
protected characteristic that are connected to that characteristic; 


(b) take steps to meet the needs of persons who share a relevant protected 
characteristic that are different from the needs of persons who do not share it; 


(c) encourage persons who share a relevant protected characteristic to participate in 
public life or in any other activity in which participation by such persons is 
disproportionately low. 


(4) The steps involved in meeting the needs of disabled persons that are different from the 
needs of persons who are not disabled include, in particular, steps to take account of 
disabled persons’ disabilities. 


(5) Having due regard to the need to foster good relations between persons who share a 
relevant protected characteristic and persons who do not share it involves having due 
regard, in particular, to the need to— 


(a) tackle prejudice, and (b) promote understanding. 


(6) Compliance with the duties in this section may involve treating some persons more 
favourably than others; but that is not to be taken as permitting conduct that would 
otherwise be prohibited by or under this Act. 


There is additional Equalities legislation in the nations, including (for instance) Section 75 of 
the Northern Ireland Act (1998) which places a statutory duty on public authorities to have 
due regard to the need to promote equality of opportunity between persons of different 
religious belief, political opinion, racial group, age, marital status or sexual orientation; 
between men and women generally; and between persons with a disability and persons 
without. 


Question 2a. Guidance on submissions: Part 2: Submissions — is the guidance clear? 


In our response to this question we suggest a clarification to how Open Access is framed, to 
highlight the importance of considering user-needs. 


Question 2a. 


We are pleased to see the efforts made to encourage interdisciplinarity and collaboration, 
inside and outside academia (e.g. 106 d), and the guidance that this should be addressed 
within the Environment Template. 


We were also encouraged by the commitment to encouraging Open Access. 


However, we are concerned that the guidance, as it stands, suggests that simply making 
research outputs ‘open and accessible’ is sufficient. We would like to see the guidance 
reference the importance of considering ‘user needs’ when developing an open access 
strategy. 


A focus on user needs is considered essential in the design of any intervention of this 
nature. The government’s Digital Service Standard provides a robust methodology to 
help inform how Open Access is approached (https://www.gov.uk/service-manual/user- 
research/start-by-learning-user-needs). The guidance pithily explains why this matters: 


‘When designing a government service, always start by learning about the people who 
will use it. If you don’t understand who they are or what they need from your service, you 
can’t build the right thing’. 


We advise that paragraph 108 is amended to emphasise the need for efforts to realise 
‘openness’ are informed by a robust understanding of the needs of potential users and 
beneficiaries. We suggest the following change: 


‘The four UK HE funding bodies believe that the outputs of publicly funded research should 


be freely accessible and widely available, and that consideration of user needs should 
inform how they are made available to potential beneficiaries’ 


The full potential of open access will not be realised without this encouragement: just 
making it ‘open’ does not mean that people will find research or choose to use it. Given 
that one of the core purposes of the REF is to ‘create a strong performance incentive for 
HEls and individual researchers’ (p.9) we think that this clarification is important. 


Question 13a. Guidance on submissions: Part 3, Section 3: Impact (REF3) 


In our response to this question we focus attention on the definition of impact and suggest 
what we believe to be a more coherent and constructive definition. This differentiates 
between different types of indicators of impact (structure, process and outcome), in line with 
good practice in other sectors. 


Question 13a 

There is some lack of consistency in how impact is defined in various parts of the guidance 
(139, 292) and draft criteria (277; Annex A). A variety of possible indicators are offered, 
but these lack coherence. Greater clarity could be achieved if these were organised more 
systematically using established approaches to evaluation, for instance in the health 
sector. 


Here, useful distinctions are drawn between: 
e ‘Structure’ indicators: e.g. policies, systems and infrastructure 
e ‘Process’ indicators: e.g. performance, activity 
e ‘Outcome’ indicators: e.g. skills, attitudes, understanding, behaviour, well-being 


These are not currently clearly differentiated in the guidance documents. If taken up, this 
could also inform the definition of impact, as follows: 


Impact includes, but is not limited to, an effect on, change or benefit to: 
e the activity, opportunity, performance, policy, practice or process ( structure 
and process indicators) 
e of an audience, beneficiary, community, constituency, organisation or 


individuals 

e jin any geographic location whether locally, regionally, nationally or 
internationally 

e affecting attitudes, awareness, behaviour, capacity or understanding (outcome 
indicators) 


There is also some lack of clarity about the extent to which ‘pathways’ to impact, when 
described in case studies, are considered by panels in making their judgements of the 
quality of case studies. We suggest that the following might be added to paragraph 318: 


Impact can happen serendipitously, but in many cases it is realised through well planned 
‘pathways’ which maximise purposeful interaction between research users and 
researchers. Such pathways can involve users in the development and shaping of the 
research, and/or involve them actively in its application. While panels’ primary focus will 
be on the outcomes realised by this kind of interaction, due consideration will be given to 
the contribution of ‘pathway’ activities when these are described. 


Background note to Q 13a: A useful overview of the use of structure, process and outcome 


indicators can be accessed here: https://improvement.nhs.uk/documents/2135/measuring- 
quality-care-model.pdf 


Here we argue that a reference to Equality and Diversity is added to the prompts about 
impact strategy 


Question 15a 

In line with our response to question 1a on the Draft Guidance, we recommend that 
paragraph 351 b is amended as follows, to include a prompt to consider equality and 
diversity in relation to impact strategy: 


Strategy: the institution’s strategy for research and enabling impact (including integrity, 


open research, considerations of equality and diversity and structures to support 


interdisciplinary research) in the assessment period and for the next five year period. 


Guidance on Submissions: further comments 
16. Please provide any further comments on the 'Guidance on submissions’, including 
Annexes A-M 


Here we make specific suggestions about how the impact case study and environment 
templates should be changed to foreground the importance of Equality and Diversity 


Question 16 
In line with our response to Q1a we suggest changes to the case study and environment 
templates, to reflect equality and diversity (E&D) considerations. We suggest that a 
reference to E&D is added to Section 4 of the Case Study template (Annex G): 
e Details of the beneficiaries [] affected or impacted on. If relevant, include details 
of how considerations of equality and diversity informed the engagement with 
beneficiaries. 


We suggest the following changes to the Unit-level Environment template (Annex l): 
Section 2. People 
e This section should provide evidence about staffing strategy [] and evidence of how 
the submitting unit supports and promotes E&D in research careers. 


Section 4. Collaboration and contribution to the research base, economy and society 
e This section should provide information about the submitted unit’s research 
collaborations, networks and partnerships, including relationships with key 
research users, beneficiaries or audiences; the wider activities and contributions to 
the research base, economy and society; and how the submitting unit supports 


and promotes equality and diversity through these collaborations and 
contributions. 


We suggest the following changes to the Institution-level Environment template (Annex 
H): 


Strategy: The institution’s strategy for research and enabling impact (including integrity, 
approach to equality and diversity, open research, and structures to support 
interdisciplinary research). 


In line with our response to question 13a we suggest that section 4 of the case study 
template should be amended to emphasise the importance of user engagement: 


A clear explanation of the process or means through which the research led to, 
underpinned or made a contribution to the impact (for example, how users were engaged 
in the research process, how the research was disseminated, how it came to influence 
users or beneficiaries, or how it came to be exploited, taken up or applied). 


DRAFT PANEL CRITERIA AND WORKING METHODS 


Part 3, Section 3: Impact 

4c. Please comment on the criteria in 'Part 3, Section 3: Impact’, in particular on: 
- where further clarification is required 
- where refinements could be made 
- whether there are areas where more consistency across panels could be 
achieved 
- whether there are differences between the disciplines that justify further 
differentiation between the main panel criteria. (300 word limit) 


Here the word limit restriction is particularly challenging. We flesh out our argument with 
more detailed suggestions in Annex 2, and provide more detail in our response to Q.8a 


Question 4c 

We are glad to see that submitting impacts arising from public engagement (PE) is 
encouraged but the guidance could be clearer. Firstly, PE is framed primarily as a route to 
realising individual understanding and learning. While this is significant, there are many 
other ways in which engaging the public in research can create impact, for instance in 
influencing policy and practice through public dialogue or co-production. 


Secondly, the guidance (especially Annex A) limits public engagement to one of the areas 
of impact (understanding) and fails to make explicit its potential contribution to the 
others, for instance to health (through patient involvement) or to areas like policy or 
business. We have published a modified version of Annex A to illustrate how this could be 
realised, which includes examples of how impacts arising from public engagement can be 
evidenced. We provide more detail in our response to Q8a. 


We recommend that the draft criteria makes more explicit reference to the role of 
evaluation as a mechanism for securing convincing evidence, and suggest the following 
wording be added to the guidance (paragraph 296): 


The use of robust evaluation methods and the provision of evidence arising from 
evaluation provides support for the impacts claimed. Assessors will bear this in mind when 
considering the evidence of impact that is cited in the case study. 


Evidence might result from the researcher / team undertaking their own evaluation (which 
could be referenced in section 4 of the case study); or commissioning an external 
evaluation or drawing on evaluations conducted by one of their partners (which can be 
referenced in section 5 of the case study template.) A list of evaluation frameworks and 
tools, and the types of evidence they can provide, is available on the NCCPE website. 


Background notes to Q 4c 


Annex 2 provides our rationale for amending Annex A: Examples of Indicators and Impact 


Panel criteria and working methods: Part 5: Panel working methods 

7c. Please comment on the criteria in ‘Part 5: Panel working methods’, in particular on: - 
where further clarification is required or where refinements could be made. (300 word 
limit) 


We have concerns about the proposed approach to assessing and moderating impact case 
studies. We suggest that a higher number of assessors for each case study is set (a minimum 
of three, with at least two research users), and that there is a clear process in place to ensure 
that those assessors are provided with the relevant information from the Environment 
template to inform their judgements. 


Question 7c 
We have some concerns about the rigour of the assessment process as outlined for 
impact case studies: 


‘Each impact case study will be allocated to at least one academic member and one user 
member or assessor, wherever practicable. User assessors will be allocated impact case 
studies, and may be allocated relevant parts of the environment template. User members 
may — in addition to impact case studies — be allocated whole environment templates 
and/or outputs in particular areas where they are willing and have appropriate expertise 
to assess them (369)’. 


We would like to see at least two research users assessing each impact case study, as a 
matter of course, alongside one academic. 


More clarity on how the scores will be moderated within and across panels would be 
helpful, to build confidence in the process from HEI and user communities. 


We consider it essential that all readers of impact case studies are provided with the 
relevant contextual information provided within the Environment templates, as this is a 
vital source of evidence which should inform their judgements. It is not clear from the 
draft guidance if and how this will be achieved, and we recommend that this is clarified. It 
would be helpful if the guidance explained more clearly how the evidence from the 
Environment section will be used to inform the assessment of case studies. 


The NCCPE would be delighted to provide training or guidance to panel members on 
assessing impacts arising from public engagement. Having one academic and one 
research user from each panel attending such training (perhaps in Main Panel clusters) 
could help ensure consistency across the panels. 


8a. Overall, the ‘Panel criteria and working methods’ achieves an appropriate balance 
between consistency and allowing for discipline-based differences between the panels 


We have used this final ‘catch all’ question to provide more detail of our proposed changes 
to Annex A. 


Question 8a 


Annex A provides a helpful articulation of the panels’ collective expectations. We suggest 
two ways in which the table could be made more consistent. 


Firstly, we suggest that differentiating between ‘structure’, ‘process’ and ‘outcome’ 
indicators would help, and that examples might be given of each to frame the table, and 
then within each Area of Impact. For instance: 


Structure indicators: 

e Evidence of influence on policies, guidelines, standards or regulations 

e Evidence of influence on decision making and accountability arrangements 

e Evidence of influence on the strategic objectives of an organisation 

e Evidence of adoption of regulations and routines to enhance social responsibility, 
ethical practice, equality and diversity and sustainability 


Process indicators: 

e Evidence adoption of new or changed methods, models, techniques and approaches 
(e.g. enhanced co-production; changes to resource-use) 

e Evidence of influence on project activity and operations, product development and/or 
service delivery 

e Evidence of challenge to ‘received wisdom’ or established paradigms and assumptions 

e Evidence of influence on capacity building activity 

e Evidence of widened participation and access (e.g. by marginalized, under-engaged 
and/or diverse audiences 


Outcome indicators 

e Evidence of increased awareness and understanding 
e Evidence of increased skills 

e Evidence of increased empathy or tolerance 

e Evidence of increased creativity 

e Evidence of enhanced patient outcomes 


The first 8 of the ‘impact areas’ in Annex A have a particular logic: each identifies a broad 
area of common purpose that motivates relatively discrete professional communities to 
realise impact. The final area (Understanding) doesn’t follow the same logic, describing 
instead various individual outcomes, and not referencing the activity of any professional 


communities. A more coherent framing of the areas of impact could be achieved if this 
ninth area of impact was revised. We suggest that it might be split into two broad areas: 
e Impacts on education, lifelong learning and skills 
e Impacts on citizenship and community engagement 


Annex 1: References to public engagement in the draft guidance 
There are four explicit references to public engagement in the two documents: 


Draft Panel Guidance 


318. There are many ways in which research may have underpinned impact, including but 
not limited to: 


Impacts on, for example, public awareness, attitudes, understanding or behaviour 
that arose from engaging the public with research. In these cases, the submitting 
unit must show that the engagement activity was, at least in part, based on the 
submitted unit’s research and drew materially and distinctly upon it. Further 
guidance and examples are set out in the ‘Panel criteria’, Annex A. 


Draft panel criteria 


274. The panels also acknowledge that there are multiple and diverse pathways through 
which research achieves impact. Impact may be the result of individual or collective 
research (or a combination of these) within or between a range of organisations, within 
higher education and beyond, including collaboration beyond the UK. The associated 
impact may be achieved by a variety of possible models: from individuals, to inter- 
institutional groups, to groups including both academic and non-academic participants. 
The relationship between research and impact can be indirect or non-linear. The impact 
of research may be foreseen or unforeseen. It can emerge as an end product, but can also 
be demonstrated during the research process. Impact takes place through a wide variety 
of mechanisms. It may effect change or enrichment for local, national or international 
communities, groups or individuals. Consequently, public engagement may be an 
important feature of many case studies, as the mechanism by which the impact claimed 
has been achieved. 


288. Engaging the public with the submitting unit’s research (for example, through citizen 
science, patient and public involvement in health, or through public and community 
engagement), is an activity that may lead to impact. Sub-panels will welcome, and assess 
equitably, case studies describing impacts achieved through public engagement, either as 
the main impact described or as one facet of a wider range of impacts. Panels expect that 
case studies based on public engagement will demonstrate both reach (e.g. through 
audience or participant figures) and significance, and will take both into account when 
assessing the impacts. Examples of impacts arising from public engagement can be found 
as part of Table 1 (Annex A). 


Annex A: Examples of impact achieved through public engagement are integrated into 
the different areas of impact in Table 1. More detailed advice on achieving and evidencing 


impact through public engagement can be found on the website of the National 
Coordinating Centre for Public Engagement: http://www.publicengagement.ac.uk/. 


4. Examples are also provided of impact evaluation frameworks used outside higher 
education. Impact partners may also have their own evaluation frameworks that could be 


drawn upon to evidence impact. 


Annex 2: reframing the Examples of Indicators and Impact (Annex A, Draft Panel Criteria) 


This annex ‘walks through’ our suggestions for how the table of indictors might be made more coherent and consistent, and how impacts arising from public 
engagement might be incorporated across the different impact domains 


Background notes to Q4c: structure, process and outcome indicators 


We reviewed Annex A to establish whether a more consistent and coherent framing of the proposed indicators might be offered. 


In doing so we identified how each provided, to a greater or lesser extent, examples of structure, process and outcome indicators. The table below lists 
examples, organised under the three headings. We suggest that the guidance might use this generic framing to provide a more coherent approach, and that 
each Impact Area could usefully provide examples of each pertinent to the area. 


Impact domain 


Structure indicators 


Process indicators 


Outcome indicators 


e Creativity & culture 
e Environment 

e Social welfare 

e Production 

e Commerce and the 


economy 

e Practitioners & prof 
services 

e Public policy, law & 
services 


e Health, wellbeing & 
animal welfare 

e Understanding, learning 
and participation 


e Evidence of influence on policies, 


guidelines, standards or 
regulations pertaining to each 
domain 


e Evidence of influence on decision 


making and accountability 


arrangements (e.g. governance 


arrangements) 

e Evidence of influence on the 
strategic objectives of an 
organisation 

e Evidence of influence on 
arrangements to support 
collaboration and partnership 


working (e.g. formal partnership 


arrangements) 

e Evidence of adoption of 
regulations and routines to 
enhance social responsibility, 
ethical practice, equality and 
diversity and sustainability 

e Evidence of positive 
environmental benefits 


e Evidence adoption of new or changed methods, models, 
techniques and approaches (e.g. enhanced co-production; 
changes to resource-use) 

e Evidence of influence on project activity and operations, 
product development and/or service delivery 

e Evidence of challenge to ‘received wisdom’ or established 
paradigms and assumptions 

e Evidence of influence on capacity building activity (e.g. CPD; 
curricula; teaching methods) 

e Evidence of changes to working practices, e.g. through 
increased reflective practice 

e Evidence of influence on decision-making outcomes (e.g. 
planning decisions) 

e Evidence of changes to extent, quality and longevity of 
collaboration 

e Evidence of changes to management practices 

e Evidence of decision making processes being influenced 

e Evidence of changes to progression opportunities for 
individuals with protected characteristics 

e — Evidence of improved access to resources or opportunities 

e Evidence of uptake of research in public debate and 
discourse 

e Evidence of the identification and mitigation of risks, and 
hazards 

e — Evidence of influence on consideration of ethics and values 

e — Evidence of reduced costs, greater profitability, enhanced 
efficiency or productivity 

e — Evidence of improved accountability / monitoring 


e Evidence of increased awareness and 
understanding 

e Evidence of increased skills 

e Evidence of increased empathy or 
tolerance 

e Evidence of increased creativity 

e Evidence of increased educational 
attainment 

e Evidence of increased confidence 

e Evidence of increased motivation 

e Evidence of increased trust 

e Evidence of enhanced health and 
wellbeing 

e Evidence of enhanced quality of life 

e — Evidence of influence on social mobility 

e Evidence of improved equality and 
diversity outcomes 

e Evidence of strengthened or extended 
networks 

e Evidence of changed behaviours 

e Evidence of increased ‘user’ satisfaction 
with products or services 

e Evidence of enhanced patient outcomes 


Background notes to Q4c: the impact areas 


We reviewed the 9 ‘impact areas’ in Annex A and identified that the first 8 had a particular logic: each identified a broad area of common purpose 

(‘motivation’) that mobilises relatively discrete professional communities to realise impact. This is represented in the table below. We noted that the final 
area (Understanding, Learning and Participation) didn’t follow the same logic, describing instead various individual outcomes, and not referencing the activity 
of any professional communities. We argue that a more coherent framing of the 9 areas of impact could be achieved if this ninth area of impact was revised. 
We suggest that it might be split into two broad areas (shown on the next page): 


e Impacts on Education, lifelong learning and skills 
e Impacts on citizenship and community engagement 


Area of impact / domain 


Motivation 


Professional communities 


Impacts on creativity, culture and society 


To stimulate participation in the arts and 
culture; to support creative expression 


People working in arts and cultural sector 


Impacts on the environment 


To enhance the environment, address climate 
change, stewardship, conservation 


People working in all areas of society, committed to positive impacts on the 
environment 


Impact on social welfare 


To address poverty and inequality and other 
social challenges 


People working in social care, and to address poverty social exclusion and social 
mobility 


Impacts on production 


To create new, or improve existing products 
and production processes 


People working in manufacturing and on the production of new products 


Impacts on commerce and the economy 


To generate wealth, create efficiencies, 
improve financial regulation 


People working in financial services and regulation, and in businesses with a ‘bottom 
line’ 


Impacts on practitioners & professional services 


To enhance professional practice & deliver 
high-quality services that meet ‘users’ needs 


People working to deliver high quality services, in any sector 


Impacts on public policy, law and services 


To create effective, just and fair polices, 
regulations and laws 


Government (national, devolved and local); regulators and law makers 


Impacts on health, wellbeing & animal welfare 


To improve human and animal health and 
wellbeing 


People working in & allied to healthcare, social care or animal welfare to deliver 
services; companies supplying products & services, carers & community initiatives 


Impacts on understanding, learning & 
participation 


To inspire individual learning and 
participation 


We are suggesting that this area should be re-framed — see next page 


Background notes to Q4c: Revised Impact Domains 


Area of impact / domain 


Motivation 


Professional communities 


Impacts on creativity, culture and society 


To stimulate participation in the arts and 
culture; to support creative expression 


People working in arts and cultural sector 


Impacts on the environment 


To enhance the environment, address climate 
change, stewardship, conservation 


People working in all areas of society, committed to positive impacts on the 
environment 


Impact on social welfare 


To address poverty and inequality and other 
social challenges 


People working in social care, and to address poverty social exclusion and social 
mobility 


Impacts on production 


To create new, or improve existing products 
and production processes 


People working in manufacturing and on the production of new products 


Impacts on commerce and the economy 


To generate wealth, create efficiencies, 
improve financial regulation 


People working in financial services and regulation, and in businesses with a ‘bottom 
line’ 


Impacts on practitioners & professional services 


To enhance professional practice & deliver 
high-quality services that meet ‘users’ needs 


People working to deliver high quality services, in any sector 


Impacts on public policy, law and services 


To create effective, just and fair polices, 
regulations and laws 


Government (national, devolved and local); regulators and law makers 


Impacts on health, wellbeing & animal welfare 


To improve human and animal health and 
wellbeing 


People working in & allied to healthcare, social care or animal welfare to deliver 
services; companies supplying products & services, carers & community initiatives 


Impacts on education and lifelong learning 


To stimulate lifelong learning and educational 
achievement 


People working in formal and informal education 


Impacts on citizenship & community 
engagement 


To increase participation in civil society, 
public engagement in decision-making and 
debate 


People working in community development and citizenship 


We then considered how best to frame the potential contribution of impacts arising from public engagement. Arising from our review of REF 2014 case 
studies, we wanted to demonstrate how engaging the public can contribute value in all areas of impact: from the environment to social welfare, policy 
making, health and the economy. We added examples of how the public can be engaged, with what kinds of results, to each of the 10 areas, as demonstrated 


on the next page 


Background notes to Q4c: Updated table 


This table attempts to provide a more consistent and coherent framing of the areas of impact, and to model how PE contributes to each 


Area of impact 


Motivation 


Professional communities 


Public roles 


Contribution of public engagement? 


Impacts on creativity, 
culture and society 


To stimulate participation in 
the arts and culture; to 
support creative expression 


People working in arts and cultural 
sector 


Audiences; participants; 
communities of place and 
interest; volunteers; 
supporters 


Stimulate involvement in arts and culture; widen 
participation and representation; support 
development of services better tuned to users 


Impacts on the 
environment 


To enhance the environment, 
address climate change, 
stewardship, conservation 


People working in all areas of 
society, committed to positive 
impacts on the environment 


Citizen scientists; activists; 
volunteers; communities of 
place 


Stimulate behaviour change; build understanding and 
awareness; contribute to public debate 


Impact on social welfare 


To address poverty and 
inequality and other social 
challenges 


People working in social care, and 
to address poverty social exclusion 
and social mobility 


Citizens; community 
members; carers 


Contribute to public debate; support development of 
services better tuned to users; inform policy 
development; address behaviours 


Impacts on production 


To create new, or improve 
existing products and 
production processes 


People working in manufacturing 
and on the production of new 
products 


Consumers; employees; union 
members 


Hold to account for business practices; contribute to 
shaping of new products so they better meet 
customer needs 


Impacts on commerce 
and the economy 


To generate wealth, create 
efficiencies, improve financial 
regulation 


People working in financial services 
and regulation, and in businesses 
with a ‘bottom line’ 


Consumers; customers; 
employees; communities of 
place; citizens 


Hold to account for business practices; contribute to 
shaping of new products so they better meet 
customer needs; stimulate innovation 


Impacts on practitioners 
& professional services 


To enhance professional 
practice & deliver high-quality 
services that meet ‘users’ 
needs 


People working to deliver high 
quality services, in any sector 


Service users; customers; 
members; trustees and 
governors; clients 


Hold to account for quality of service; contribute to 
shaping of new services so they better meet user 
needs; embed reflective practices 


Impacts on public policy, 
law and services 


To create effective, just and 
fair polices, regulations and 
laws 


Government (national, devolved 
and local); regulators and law 
makers 


Citizens; consumers; 
community members; voters 


Contribute to public debate; support development of 
policies, governance and laws; inform policy 
development through dialogue 


Impacts on health, 
wellbeing & animal 
welfare 


To improve human and 
animal health and wellbeing 


People working in & allied to 
healthcare, social care or animal 
welfare; companies supplying 
products & services, carers & 


Patients; carers; parents; pet 
owners 


Stimulate awareness and understanding; address 
behaviours; enhance services through patient 
involvement; improved prevention, treatment and 
support 


community initiatives 
Impacts on education To stimulate lifelong learning People working in formal and Learners; community Stimulate take up of lifelong learning; encourage 
and lifelong learning and educational achievement | informal education members; pupils; parents participation in civic life; enhance services through 


user engagement 


Impacts on citizenship & 
community engagement 


To increase participation in 
civil society, public 
engagement in decision- 
making and debate 


People working in community 
development and citizenship 


Citizens; community 
members; communities of 
place 


Improve participation in civil society, informal 
engagement in decision-making and involvement in 
public/online discussion 


Background notes to Q4c: Structure, process and outcome indicators applied to impacts arising from public engagement 


Finally, we revisited the indicators of impact listed in the different Areas of Impact, and proposed examples of indicators that might be utilised to capture 


impacts arising from public engagement 


Structure indicators 


Process indicators 


Outcome indicators 


e Evidence of research process involving the 
public in the shaping of new policies, 
guidelines and regulations 

e Evidence of research process resulting in 
public involvement becoming embedded 
in governance and accountability 
arrangements 

e Evidence of research process ensuring 
public ‘voice’ holds organisations to 
account 


e Evidence of research process leading to enhanced 
public involvement in the shaping and delivery of 
services 

e — Evidence of discussion and debate being stimulated in 
the public sphere 

e Evidence of individual / group decision making being 
influence by research 

e — Evidence of the increased uptake of lifelong learning 
Evidence of increased public participation in the 
different domains 

¢ Evidence of widened participation and access (e.g. by 
marginalized, under-engaged and/or diverse 
audiences 


Evidence of how specific individuals or groups have been directly affected 
in all indicator areas 


Evidence of increased awareness and understanding 
Evidence of increased skills 

Evidence of increased empathy or tolerance 

Evidence of increased creativity 

Evidence of increased educational attainment 
Evidence of increased confidence 

Evidence of increased motivation 

Evidence of increased trust 

Evidence of enhanced health and wellbeing 

Evidence of enhanced quality of life 

Evidence of influence on social mobility 

Evidence of improved equality and diversity outcomes 
Evidence of strengthened or extended networks 
Evidence of changed behaviours 

Evidence of increased ‘user’ satisfaction with products or services 
Evidence of enhanced patient outcomes 


Centre for 


National 
| | | Co-ordinating 
| Public Engagement 


The National Co-ordinating Centre for Public Engagement (NCCPE) is 
internationally recognised for its work supporting and inspiring universities 
to engage with the public. 

We work to change perspectives, promote innovation, and nurture and 
celebrate excellence. We also champion meaningful engagement that makes 
a real and valued difference to people’s lives. 


The NCCPE is supported by the UK Higher Education Councils, Research 
Councils UK and Wellcome, and has been hosted by the University of Bristol 
and the University of the West of England since it was established in 2008. 
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